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Abstract. The spectral decomposition for an explicit second-order differential operator T is 
determined. The spectrum consists of a continuous part with multiplicity two, a continuous 
part with multiplicity one, and a finite discrete part with multiplicity one. The spectral analysis 
gives rise to a generalized Fourier transform with an explicit hypergeometric function as a kernel. 
Using Jacobi polynomials the operator T can also be realized as a five-diagonal operator, hence 
leading to orthogonality relations for 2 X 2-matrix- valued polynomials. These matrix- valued 
polynomials can be considered as matrix- valued generalizations of Wilson polynomials. 

1. Introduction 

It is well-known that a three-term recurrence relation 

Ap„(A) = a„p„+i(A) + 6„_p„(A) + a„_ip„_i(A), n = 0, 1,2, . . . , 

with a_i = 0, can be solved using orthogonal polynomials. A generalization of this is obtained 
by Duran and Van Assche [5], who showed a 2N + 1-term recurrence relation can be solved using 
N X A^-matrix-valued orthogonal polynomials. Motivated by this result and previous work by 
Ismail and the second author [12], [13], a method is presented by Ismail and the authors [8] to 
obtain orthogonality relations for 2 x 2-matrix-valued orthogonal polynomials from an operator T 
on a Hilbert space T-L of functions. The operator T must satisfy the following conditions: 

(i) T is self-adjoint; 

(ii) there exists a weighted Hilbert space L^(V) and a unitary operator U :% ^ so that 
UT = MJ7, where M is the multiplication operator on L^(V); 

(iii) there exists an orthonormal basis {/n}^o '^^ there exist sequences (a„)5^Q, (6n)^0! 
(cn)iJLo of numbers with a„ > and c„ e M for all n G N, such that 



Tfn — 0,nfn+2 + bnfn+1 + Cnfn + bn-lfn-1 + fln-2/n-2, 

where we assume a_i = a_2 = b-i — 0. 

In [8] two explicit examples are worked out, where the operator T is, besides a five-term op- 
erator, also realized as the second-order ^-difference operator corresponding to well-known q- 
hypergeometric orthogonal polynomials. Thus, the unitary operator U is the integral transform 
with the corresponding orthogonal polynomials as a kernel. This leads to complicated, but ex- 
plicit, orthogonality relations for certain matrix-valued polynomials defined by an explicit matrix 
three- term recurrence relation. We note that the explicit weight function differs structurally from 
the usually considered weight functions for matrix-valued orthogonal polynomials consisting of a 
matrix-deformation of a classical weight. 

In this paper we apply the method from [8] with the second-order differential operator T — 
rpia,P;^) defined by 

(1.1) T = (1 - + (1 - x^)[P -a-{a + l3 + 4)x]^ + h^,^ - [a + (3 + 3)2](1 - x^). 

ax^ ax 4 

Here a, (3 > —1 and k e M>o U iM>o. The differential operator T is closely related to the second- 
order differential operator to which the Jacobi polynomials are eigenfunctions. It should be noted 
that T raises the degree of a polynomial by 2, so there are no polynomial eigenfunctions. We 
will show that the differential operator T, considered as an unbounded operator on a weighted 
L^-space, satisfies conditions (i)-(iii) given above. An interesting problem here is that T does not 
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correspond to orthogonal polynomials or to a known unitary integral transform such as the Jacobi 
function transform [16]. 

The unitary operator U needed in condition (ii) is given by an explicit integral transform 
which is obtained from spectral analysis of T. The spectrum of T consists of a continuous part with 
multiplicity two, a continuous part with multiplicity one, and a (possibly empty) finite discrete 
part of multiplicity one. As a result, the integral transform has a hypergeometric kernel which 
is partly C^-valued and partly C-valued. There are several (but not very many) hypergeometric 
integral transforms with C^-valued kernels available in the literature, sec e.g. [18], [9], [15, Exer- 
cise (4.4.11)], see also [7] for an example with basic hypergeometric functions. To the best of our 
knowledge all known examples can be considered as nonpolynomial extensions of hypergeometric 
orthogonal polynomials, in the sense that the corresponding kernels are eigenfunctions of a differ- 
ential/difference operator that also has orthogonal polynomials as eigenfunctions. For example, 
Neretin's C^-valued 2F1- integral transform [18] generalizes the Jacobi polynomials. The integral 
transform T we consider in this paper, however, docs not seem to generalize a family of orthogonal 
polynomials, although in a special case it is can be considered as a nonpolynomial extension of 
two different one-parameter families of Jacobi polynomials. Furthermore, other hypergeometric 
integral transforms and hypergeometric orthogonal polynomials correspond to a bispectral prob- 
lem, see e.g. [10], which can always be related directly to contiguous relations for hypergeometric 
functions. From the explicit expressions as hypergeometric functions for the kernel of it is 
unclear wether T is also related to a bispectral problem. 

In a special case the 2 x 2-matrix- valued orthogonal polynomials we obtain can be diagonalized. 
In this case the orthogonality relations correspond to orthogonality relations for two subfamilies of 
Wilson polynomials [20] . This is why we consider our matrix- valued polynomials as generalizations 
of (subfamilies of) Wilson polynomials. 

The organization of this paper is as follow. In Section 2 we introduce the integral transform T 
and show that the differential operator T (1.1) satisfies conditions (i) and (ii). The proofs for this 
section are given separately in Section 4, where the spectral analysis of T is carried out, which 
can be quite technical at certain points. In Section 3 we realize T as a five-diagonal operator on a 
basis consisting of Jacobi polynomials, so that condition (iii) is also satisfied. The corresponding 
five-term recurrence relation is equivalent to a matrix three-term recurrence relation that defines 
2 X 2-matrix-valued orthogonal polynomials P„ for which the orthogonality relations are deter- 
mined. We also consider briefly the special case a = (3, in which case the integral transform T 
reduces to two Jacobi function transforms and the orthogonality relations for P„ correspond to 
the orthogonality relations for certain Wilson polynomials. Finally, in Section 4 eigenfunctions of 
T are given, which are needed for the spectral decomposition of T. The spectral decomposition 
leads to a proof of the unitarity of the integral transform J-, and to an explicit formula for its 
inverse. 

Notations. We write N for the set of nonnegative integers. We use standard notations for 
hypergeometric functions, as in e.g. [2, 11]. For products of F-functions and of shifted factorials 
we use the shorthand notations 

r(ai, 02, . . . , a„) = r(ai)r(a2) • • • T{a„), 

(ai, a2, . . . , an)k = (ai)fe(a2)fe • • • {an)k- 

2. Spectral analysis and a hypergeometric function transform 

In this section we describe the spectral analysis of the operator T defined by (1.1). The spectral 
decomposition is given by an integral transform with certain hypergeometric 2F1- functions as a 
kernel which is interesting in its own right. The proofs for this section are postponed until Section 4. 

Let a, /3 > — 1 be fixed, and let w^"'^^ be the Jacobi weight function on [—1, 1] given by 

(2.1) w^-'^\x) = 2— ^-^ ^y^"Y + 'i. (l - xni + xf. 

I (a + I, p + I) 
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The corresponding inner product is denoted by (•,•), 

(/,5)= j J{x)W)w^'''^\x)dx. 

The weight is normahzcd such that (1,1) = 1. We denote hj % = the corresponding 

weighted L^-space; % = L^{{~l,l),w^"-'^\x)dx). To stress the dependence on the parameters a 
and /?, we will sometimes denote the inner product in "H^"'^) by (•, ■)a,fi- Let us remark that the 
substitution x n> -x sends T^"''^''") to T^^'"'''), and 'H^"''^) to 'H^l^'"^ So without loss of generality 
we may assume P > a, which we do from here on. 

Wc consider T as an unbounded operator on H. The domain Dq for T is described in Section 
4.2, wlicrc the following result is proved. 

Proposition 2.1. The operator (T, 2?o) has a unique self-adjoint extension. 

We denote the extension of T again by T. The spectral analysis of T will be described by 
the integral transform T mapping functions in H (under suitable conditions) to functions in the 
Hilbert space L^{V). We first introduce the latter space. 

Let fii, O2 C M be given by 

= (-(/3 + l)^-(a + l)2) and ^2 = ( - 00, -(/3 + 1)^). 



We set 



(2.2) 



5;, =i^-A- (a; + l)2, AefiiUOa, 



r]x = iV-\-{f3 + l)^ A e O2, 



d{\) = VA + (a + l)2, AeC\(QiUf22), 
rj{X) = V^ + (/3 + l)2, AeC\Q2. 

Here y/^ denotes the principal branch of the square root. For n e N, we define A„ e C as the 

solution of 



(2.3) (5(A) + r?(A) = ^A + (a + 1)2 + ^A + (/3 + 1)2 = -2n - 1 + k. 

We define the finite set CI4 by 

= {A„ I n e N and n < ^(k - 1)}, 

i.e., rtd consists of the real solutions of (2.3). Note that fi^ = if k < 1 or k e zK>o. The number 
An € has the explicit expression 

We will denote by a the set f22 U fii U O^. Theorem 2.2 will show that a is the spectrum of T. 
Next we introduce the weight functions that we need to define L'^{V). First we define 

(2 4) c(x- v) = r{l + y,-x) 

^ n\{l+y-x + K),\{l + y-x-n)y 

With this function we define for A e ili 

1 



c(^a;?7(A))c(-^a;??(A)) 
For A e f22 we define the matrix-valued weight function V{\) by 

1 ^^12(A)' 



(2.6) V{X) = 



^^21 (A) 1 
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with 



(2.7) V2i(A) = 



c{-Vx;Sx) r{r]x,^{l + Sx-r]x + n),^i^ + h-Vx-i^)) 



and wi2(A) = ■i;2i(A). Finally, for A„ e Ctd we set 

/ c(r;(A);J(A)) \ 
A^tl^ry(A)c(-^(A);5(A))J 

4<5(A„) {-irr{-r^{X,,),K-n) 

-2n-l + K n! r(77(A„), i(l + 6{Xn) - r?(A„) + k), i(l + (5(A„) - r/(A„) - At)) ' 

Note here that 5(A„) - ry(A„) = ^"1^,^^^+^+"^ 

Now we are ready to define the Hilbert space L'^{V). It consists of functions that are C^-valued 
on 02 and C- valued on fli U O^. The inner product on i/^(V) is given by 



(/,fl)v = ^^ 9{XrV{X)f{X)- 



dX 
-irix 



where D = 



4r(a + ^ + 2) 



r(a + l,/3 + l) 

Next we introduce the integral transform T. For A e Oi and x e (—1, 1) we define 

' i(l + (5a + r?(A) - k), i(l + (5a + r?(A) + k) 1 + x 



l+ry(A) ' 2 

By Euler's transformation, see e.g. [2, (2.2.7)], we can replace 6x by —(5a in (2.9). Furthermore, 
we define for A e fi2 and x G (—1, 1), 

(2.10) "^'^'H— j [—) 

fl{l + dx±Vx~K),^{l + 5x±Vx + i^) 1 + 2; 

^^"^^l 1±^A 



Observe that <^a l-^) ~ 'fix (^)' again by Euler's transformation. Finally, for A„ G ild we define 



(2.11) ,...(x)=(^— j J "''■Vl + „(A„)- 2 

Now, let T be the integral transform defined by 



(2.12) (^/)(A) = <^ 



r/(^)HsiV^"''H^)^^, Aeo2, 

y" /(x)(^A(a;)w("'''^(x)dx, A e Oi, 

/ f{x)<pxAx)w^"'^\x)dx, A = A„end, 

W-l 



for all f E H such that the integrals converge. The following result says that is the required 

unitary operator U from the introduction. 

Theorem 2.2. The integral transform T extends uniquely to a unitary operator JF :'H -^^(V) 
such that TT = MT, where M : i^(V) -^^(V) is the unbounded multiplication operator. 
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Remark 2.3. In case a = /3 the spectral decomposition of T can be described using the Jacobi 

function transform [16]. To sec this, wc apply the change of variable x = tanh(t), then the 
second-order differential operator T defined by (1.1) turns into 

;s d'^ ,n , n w M K2-(a + /3 + 3)2 



df^ ^ ^ ' "-^'dt 4cosh^(t) 

I\T/~nTT /-I /-» 4-1 n 77 



For a = P, let fx be a solution of the eigenvalue equation Tfx = A/a. Now define F^{t) 



cosh5(2«+3±«)(t)/^(i), then Fx satisfies 

^ + (1 ± K) tanh(t)^ = (a + (a - 1)^ - i(l ± «)^)f±. 

Using the differential equation for Jacobi functions, sec [16, (1-1)], we now see that the spectral 
decomposition of T can be given using the Jacobi function transforms corresponding to the Jacobi 

functions 4'/'^''^ and 4;^'"*"^ 

We have an explicit inverse of the integral transform Define for x £ (—1) 1) the integral 
transform Q by 

(^^^^^^ = ^^^^''^ V'a(^))V^(A)/(A)^ 

for all functions / G i^(V) for which the above integrals converge. 



Theorem 2.4. The integral transform Q extends uniquely to an operator Q : L'^iy) — ^ W swcft 
that g = F-^. 

Theorem 2.2 and 2.4 are proved in Section 4. The following orthogonality relations are a result 
of Theorem 2.2 by considering the discrete spectrum of T. 

Corollary 2.5. Let k > 1, then the following orthogonality relations hold 

-n,K — n 1 + x 



r T? (-m.,K-m l + x\ 



l + r?(A„) ' 2 
X (1 - a;)3(5(^-)+5(A„)-2)(i ^ 1 (^(A„)+^(A„)-2) 

2^— n— m 

/or a/Z n, m € N swcft <fta< n,m< \{k—1). 

3. Matrix-valued orthogonal polynomials 

In this section we show that the differential operator T can be realized as a five-diagonal operator 

with respect to an orthonormal basis for %. Using the spectral decomposition for T this leads to 
orthogonality relations for 2 x 2-matrix- valued orthogonal polynomials. 



3.1. The five-diagonal operator. The Jacobi polynomials are defined by 

-n,n + a + p + \ 1 — x 
a+l ' 2 

For a, P > —1 they form an orthogonal basis for "H; 



/p(a,/3) p(«,/3)\ _ ^ ^(«,/3) h{a,0) _ a + P + 1 {a + 1, P + l)r 

V m ' n / — '^mn"'n ) "'n — 



2n + a + p + l {a + /3 + l)nn\ 
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The Jacobi polynomials are eigenfunctions of the Jacobi differential operator 

dx'^ ax 

^(a,/3)p^a,^!) ^ + ^ + + l)p^"./3) . 

We define r{x) = 1 — x^, then for x G (—1, 1) the polynomial r can be written as 
(3.1) rjx) = K^'":''l:y , K= 4(a+l)(/3 + l) 



w("'^){x) ' (a + /3 + 2)(a + /3 + 3)' 

The differential operator T = T(°'^5«) defined by (1.1) is related to L'-"'^'^ by 

(3.2) T("-'5;«) = M(r)(L("+i''5+i)+p), p = i - {a + p + Sf) , 

where M{r) denotes multiplication by r. 

In [8, Section 3.1] it is shown that an operator of the form (3.2) acts as a five-term operator 
on a suitable basis of H. In this case the basis consists of Jacobi polynomials. We define = 

Pn"''^V(^n"'''V^^ n e N, then {(/)„}„eN is an orthonormal basis for H^'^'^^ Wc also define 

= Pt+^''^+^^ /{h^^+^'^+^'^y/^, n G N, then {$„}„eN is an orthonormal basis for 
In order to write T explicitly as a five-diagonal operator on the basis {(f>n}neN, we need a connection 
formula between {(f>n}neN and {$„}„eN. 

Lemma 3.1. The following connection formula holds, 

(j)n = an^n + Pn^n-l + 7n*n-2, 

where 



1 / (a -I- n l)(/3 + n + l)(n + a + /3 + l)(n + a + /3 -h 2) 



V^2n + a + /3 + 2\/ (a + /3 + 2n + l)(a + /3 + 2n + 3) 



^„ = (-l)"-^ + " + + 



(a + /9 + 2n)(a + /3 + 2n + 2) ' 



7n 



2 1 / n(n- l)(a + n)(/3 + n) 



/^2n-Fa-F/3y (a + ^ + 2n - l)(a /3 2n -M) ' 
Proof. There exists an expansion 0„ ~ J2k=o ^n,k^k, where 

Since r has degree 2, it follows from the orthogonality relations for that o„,fc = for < fc < 
n — 3. 

We compute the three remaining coefiicients. The value of a„,„ follows from comparing leading 
coefiicients; a„,„ = j^p^- We have 



lc(</.„) = 2-"(n + a + /3 + 1)„ 



2n + Q + /3 + 1 {a + + l)n 
a + l3 + l n!(a + l,/3 + l)„ 



and lc(<i?„) is obtained by replacing (a, /3) by (a + 1, /3+ 1), which leads to the result for a„ = a„,„. 
For a polynomial p of degree n, let k{p) denote the coefficient of (1 — a;)"~^, then 

_ k{(j)n) - an,nk{^n) 

lc($„_i) 

We have 

k{4>n) = {-ir+'\c{4>n) 



n+i.f^ ^ 2n(a + n) 



a + ,/3 + 2n 

and k{<^n) is obtained by replacing {a,jS) by (a + l,/3 + 1), which gives the expression for /3„ 

^n,n— 1 • 
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Finally, the expression for 7„ = a„,„_2 follows from 

.il c(r^n-2) ^ ^^_i lc(^„ -2 
lc(?i„) lc((?!), 

We now use [8, Lemma 3.1] to write the differential operator T as a five-diagonal operator. 



0-71,71-2 = , . , ^ = — iV , , , ^ ■ U 



Proposition 3.2. The operator T defined by (1.1) acts as a five-diagonal operator on the basis 
{(pnjnen ofH by 

(3.3) T<f)n = an(t>n+2 + bn4>n+l + Cn(t>n + ^n-10n-l + 0„_2(/>n-2, 

wii/j coefficients given by 

On = -ftra„7„+2(A„ + p), 6„ = i^a„/J„+i(A„ + p) + ii:^„7„+i(A„+i + p), 
c„ = ifa^(A„ + p) + if/3^(A„_i + p) + /s:72(A„_2 + p), 
where A„ = — n(n + Q; + /3 + 3), -ft', p are given by (3.1), (3.2) and an, fin, are as in Lemma 3.1. 
One easily verifies that a_i — a_2 = = 0. Furthermore, we have the factorization 

A„ + p = - (n + i(a + ^ + 3 + k)) (n + ^(a + /3 + 3 - k)). 

3.2. Matrix-valued orthogonal polynomials. From Theorem 2.2 and Proposition 3.2 it follows 
that the functions fF(f)n, n e N, satisfy the five-term recurrence relation 

a(j-^„)(a) = 

( 3.4 ) 

a«(-^'/)«+2)(A) + 6„(J"(/i„+i)(A) + c„(J"0„)(A) + 6„_i(J"(/)„_i)(A) + a„„2(-^0n-2)(A), 

for almost all A e a. Furthermore, the set {F(j)n}nen is an orthonormal basis for LP'iy). We can 
determine an explicit expression for T(j)n in terms of hypergeometric functions. 

Lemma 3.3. For n e N, let Fn{S, rf) = Fn{S, r]; a, /3, k) denote the series 
^ {~n, n + a + p + l,^ia + S + l)i 
' " " IKa + l,U^ + P + v + S + 2)), 

i(l + ^ + r? + K),i(l + ^ + r?-K),i(/3 + r?+l) \ 
l + r],^{a + + r] + 5 + 2 + 2l) 'J 

with 



_1 / 2n + g + /3 + 1 (g + ^ + 1, g + 1)„ r(Q + ^ + 2, i(Q + J + 1), |(/3 + + 1)) 
""2y a + /3 + l n!(/3 + l)„ r(a + 1,^ + 1, i(a + /3 + + 5 + 2)) " 

Then, for X G a, 

FniS\,r]x) 



\r (A . , , A e O2, 

Fn{6x,v{X)), AeOi, 
LF„ (5(A), 77(A)), XeQd. 



The above sJl^-series converges absolutely if ^{a— 6+1+21) > 0, which is the case if A e f2iUf22- 
For A e fid the 3F2-series terminates. 

Proof. We compute 

X 2F,(-"'" + ;^+^+Si^)«,(-«(a.)da=. 
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Interchanging the order of summation and integration, and using the Beta-integral, we obtain 

/- =r V V " + " + '-^ + ^M:^ + + 'I + -2^'^ + + "^^"^ 

X J (l-a;)5(«+*-i)+'(l + x)^(^+''-i)+™rfa; 

_ , (-n,n + a + 3 + 1, ^(q; + (5 + 1); 

nl^Z^ lUa + l)i 

;=o m=0 ^ ^' 

^ + (5 + r? + k), i(l + J + ?7 - k), + 77 + 1))„ 



where 



m!(l + r?)„ + /3 + r? + 5 + 2)),+„ 

1 r(a + ;3 + 2, i(a + (5 + 1), + 77 + 1)) 



2 r(a + l,/3+l,i(a + /3 + r7 + 5 + 2)) 
Now the result follows from the explicit expressions for and the integral transform □ 

Prom Lemma 3.3 it follows that 



Fo{6,r]) = Do3F2 

and 

Fi{S,rj) = Di 



i(l + ,5 + 77 + k), i(l + (5 + 77 - k), i(/3 + 77 + 1) 
1 + 77, i(a + /3 + 77 + ^ + 2) 



/ i(l + ^ + 7? + Ac), i(l + 5 + 7? - Ac), + 7? + 1) 

3^2 



1 + 7?, i(Q! + /3 + 77 + (5 + 2) 



(a + /3 + 2)(a + 5 + l) ^ / i(l + ^ + 77 + k), i(l + ^ + r? - k), + r? + 1) 



(a + l)(a + ^ + 7? + 5 + 2) ' ' \^ i + r],l{a + p + r] + S + 4) 

These two functions and the five-term recurrence relation from (3.4) completely determine the 
functions J^cj^n- 

Wc define 2 x 2-matrix- valued orthogonal polynomials P„, 71 e N, by the three- term recurrence 

relations 

(3.5) AP„(A) = A„P„+i(A) + B„P„(A) + ^;_iP„_i(A), A e a, 

A _ f a2n A T3 _ fc2n ^2n 



&2n+l Cl2n+lJ ' " C2n+1 

and the initial conditions P_i(A) = and Po(A) = /. Prom the five-term recurrence relation (3.4) 
we obtain, for m € N, 

^3^^ ut;(i))(^)=^-(<j::(j)) 

'^^^-^')*\A) = P™(A)&J^!n ifAefl.. 



^.F(/.2™+i(A)V ^^'^ - \J'M^)\ 

The orthogonality relations for J^(j)n, 7i G N, can now be reformulated as orthogonality relations 
for the matrix- valued polynomials P„, see [8, Theorem 2.1]. 

Theorem 3.4. The 2 x 2-matrix-valued orthogonal polynomials Pn, tz G N, defined by (3.5) satisfy 
the orthogonality relations 

Smnl=^ f Pm{X)W2iX)Pn{Xr^ 

27rP' Jn2 -i''n\ 

+ [ ^™(A)W^i(A)P„(A)*7;(A)^ + l ^ P„(A)W^i(A)P„(A)*7Va 
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with 

W2(A) 



(^0o)(A)(J-0i)(A) |(^</'i)(A)|2 
/((J-0o)(A), (J-0o)(A))y(A) ((J-0o)(A), iT^i)iX))viX) 

l^((J-0l)(A), (J-<^o)(A))y(A) ((^-/-OlA), (^</.i)(A))y(A) 



where {x,y)v(X) = x*ViX)y. 

Remark 3.5. In [8, Proposition 3.6] a q-analog of Theorem 3.4 is considered. Tlie functions (/)„ in 
this case are the httle g'-Jacobi polynomials, and the integral transform T is simply the integral 
transform corresponding to the continuous dual q- Hahn polynomials. It would be very interesting 
to see if similar results can be obtained for other g-analogs of the Jacobi polynomials, such as big 
g'-Jacobi polynomials [1] , Askey- Wilson polynomials [3] and Ruijsenaars' ii-function [19] . 

3.3. The special case a = 13. We assume a = /3, and for convenience we also assume Sla = 0- 
In this case f^i = 0, and 5x = ri\ for all X G fl2- The spectral decomposition of T can now be 
obtained in a different way. 

The coefficient 6„ in the five-diagonal expression for T vanishes, so T reduces to a tridiagonal 
operator or Jacobi operator. Explicitly, 

T(/)n = an4>n+2 + C„(?!)„ + a„_2(/'n-2, 

with 



_ (n + ^(g + ^ + 3 + K)){n + ^(a + ^ + 3 - k)) / (n + 2)(n + l)(rt + 2a + l)(n + 2a + 2) 
~ 2n + 2a + 3 y (2n + 2a + l)(2n + 2a + 5) ' 

_ (n + 2a + l)(n + 2a + 2)(n + |(a + ^ + 3 + K)){n + ^(a + /3 + 3 - k)) 
~ (2n + 2a + l)(2n + 2a + 3) 

n{n - l)(n + \{a + /3 - 1 + + \{a + ^ - 1 - k)) 

(2n + 2a- l)(2n + 2a + l) " 

The spectral decomposition can be described with the help of the orthonormal Wilson polynomials 
[20, 2], which are defined by 



W„(x^;o,6,c, d) 



/ {a + h, a + c, a + d)n{a + b + c + d)2n 
y n\{h + c,h + d,c + d,n + a + b + c + d— 1)„ 

f —n,n + a + h + c + d — l,a + ix,a — ix 
V a + b,a + c,a + d 



If a, 6, c, d > these polynomials are orthonormal with respect to an absolutely continuous measure 
on (0, oo). 

For m e N we define 

W^ix) = Wm{i2xf- \{a + 1), i(a + 1), \{l + «), ^(1 - k)), 

W^{x) = Wm{{2xf; i(a + 1), \{a + 1), \{Z + k), i(3 - k)), 

then we obtain from the three-term recurrence relation for the Wilson polynomials 

-((a + 1)2 + x')WUx) = a2™W^,+i(a;) + C2mW^{x) + a2m-2W^_,{x), 

-((a + If + x^)W°,ix) = a2m+lW^+lix) + C2m+lW^ix) + a2m-lW^-lix). 

Let /i° and //° denote the orthogonality measures for the Wilson polynomials and Wf^. The 
unitary operator U : H — > £2(^°) ® L'^{n°), given by 



(3.7) Uct>n 



if n = 2m, 
if n = 2m -|- 1, 
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satisfies UT = MU, where M is multiplication by —{{a + 1)^ + x^). So T indeed has continuous 
spectrum (—00, — (a + 1)^) = fl2, with multiplicity 2. 

The Hilbert space can also be split up in a natural way. Prom (1.1) we see that T 

(with a = P) leaves invariant the subspaccs of even/odd functions, so wc can split H accordingly 
into T-L'^ and H". The Jacobi (Gegenbauer) polynomials 4>2m{x) are even polynomials, hence they 
form an orthonormal basis for H^, and by a quadratic transformation they can be transformed into 

multiples of P^T' ^\2x^ — l). Similarly, the odd polynomials 02m+i (a;) form an orthonormal basis 
for "H", and they can transformed into multiples of xPm'^\2x^ — 1). Obviously there are similar 
transformations for the Jacobi polynomials $2m and $2m+i- Now the operator T restricted to 
or can be treated as in [12, Section 3]. The unitary operator U is given in each case by a Jacobi 
function transform, see Remark 2.3, so that we obtain a special case of Koornwinder's formula 
[17] stating that Jacobi polynomials are mapped to Wilson polynomials by the Jacobi function 
transform. In this light, (3.6) can be considered as a matrix-analog of Koornwinder's formula. 

Remark 3.6. There exists an extension of Koornwinder's formula on the level of Wilson polynomials 
[6, Theorem 6.7]: Wilson polynomials are mapped to Wilson polynomials by the Wilson function 
transform. It would be interesting to see if there also exists a matrix-analog of this formula. 

4. Proofs for Section 2 

In this section we perform the spectral analysis of the second-order differential operator T 
defined by (1.1), considered as an unbounded operator on H. 

4.1. Eigenfunctions. The eigenvalue equation Tf\ = A/a is a second order differential equation 
with regular singular points at 1, —1 and 00, so it has hypergeometric (i.e. 2^1) solutions. We first 

determine these solutions. 

We define for A G C \ (fii U 172) the functions 

/ 1 _ a; \ - 3 ic.+5{X)+l) (/3+r,(A)+l) 



X 2F1 



2 1 \ 2 

i(l - (5(A) - r?(A) - k), \{l - 5{\) - j?(A) + k) 1 + x_ 

l^rj{\) ' 2 



X 2F1 



i(l - (5(A) + r?(A) - k), i(l - (5(A) + r/(A) + k) _ 1+^ 
1 + r?(A) ' 2 



and 



i(a+5(A)+l) -i(^+r,(A) + l) 



X 2^1 



1(1 _ 5(A) - ,y(A) - k), 1(1 - (5(A) - 7y(A) + «) 1_ 
1-(5(A) 

1 _ a.\ -5(«-'5W + l) / 1 + -5(/3+'7(A)+l) 



X 2-Fl 



> 2 / V 2 

i(l + d{X) - 77(A) - k), i(l + (5(A) - 77(A) +k) 1_ 

1 + 5{X) ' 

where 6 and 77 are defined by (2.2). From Euler's transformation for 2-Fi-series it follows that cj)^ is 
invariant under (5(A) i-)- —(5(A); similarly, t/jJ is invariant under 77(A) —77(A). Note that ip^ix) is 
obtained from 4''^ix) by the substitution (a, /?, x) i-)- (/3, a, — x), and vice versa. Note also that the 
2-Fi-series in (pf is summable at a; = 1 if JR((5(A)) > 0, and that the 2-fi-series in ipf is summable 
at a; = -1 if 5R(77(A)) > 0. 
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Remark 4.1. Prom here on we will just write S and r], instead of S{X) and ?7(A). 

Proposition 4.2. The functions <pf, ipf are solutions of the eigenvalue equation Tf = A/. 

Proof. Suppose / is a solution of the eigenvalue equation Tf = Xf. A calculation shows that if 
f{x) = (1 - a;)-3(«+*+i)(l + a;)-5('3+''+i)0(a;), then (p satisfies 

(1 - x'^)(j)"{x) + [5-rj + {S + Tj - 2)x](l}'{x) - ^{1 - rj - 5 - k)(1 - t] - S + K)(j){x) = 0. 
Now set t = ^{1 + x), then 



'2. 



1,. . . 1 



^(1 - t)^ + (1 - »?) - (1 + 7^(1 - 5 - r? - k) + -(1 - ^ - r? + K))t 



d(t) 



2' • ' • 2' 

- ^(1 - ?7 - (5 - - ry - (5 + = 0. 



4 

This is the hypergeometric differential equation (see e.g. [2, Ch.2]) with coefficients 

a= ^(l--^ -??-«), ^(l-<5 + c=l-r]. 

The 2 -Fi -functions in are well-known solutions of this differential equation, so (pxt'4'x are 

solutions of the eigenvalue equation. The proof for and tp^ is similar. □ 

For later references we need connection formulas for and 'ip^- 
Proposition 4.3. For c defined by (2.4), 

(4.1) i^fix)=c{iT,±6)^+{x)+c{-v;±5)cP^ix), 

(4.2) ^^{x) = c{d;±rj)^+{x)+c{-S;±r])iPxix)- 

Proof. This follows from a three-term transformation for 2^^i-functions, see e.g. [2, (2.3.11)]. □ 
The following identities for the c-function turn out to be useful. 

Lemma 4.4. The c-function defined by (2.4) satisfies: 

(i) c{x;y) = -f c(-j/;-x), 
(a) c(x; y)c{-x; -y) - c(x; -y)c(-x; y) = -f . 

Remark 4.5. Using the reflection equation for the F- function, Lemma 4.4(ii) is equivalent to the 
trigonometric identity 

sin(7ra;) sin(7ry) = sin(^(?/ - x + k)) sm{^{y - x - k)) - sin(^(-y - a; + k)) sin(^(-t/ - a; - «;)), 
and can also be proved in this way. 

Proof. The first identity follows from T{z + 1) = zT{z). 

For the second identity we note that Proposition 4.3 implies 

c{5]Ti) c{-S;t]) \ ^ f c{ri;5) c{-ij;S) 

c{S;-r]) c{-S;-r])J \c{r];-S) c{-r];-5) 

which in turn implies 

c{-v;-S) 



2{d; T]) 



c{m 5)c{-ri; -5) - c{r]; -S)c{-r]; 6) 
Applying the first identity to the numerator then gives 

c{v; 5)c{-ri\ -5) - c{ri\ -5)c{-r]; 5) = --, 

which proves the second identity. □ 

We also need the behavior of the eigenfunctions near the endpoints —1 and 1. 
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Lemma 4.6. For x l —1 we have 



't>xi^)=[^) {l + 0{l + x)). 

/I _ -^(«=F<5+1) , , 
i^tix)=[^-^j [1 + 0{1-X)). 



For x^ 1 we have 

■i(aT<5+l) 



Proof. This is straightforward from the cxphcit expressions as 2fi-series. □ 

Remark 4.7. Observe that the function Icp'^l^w'^"'^'^ is in L^{-1,0) if and only if ±3?(?7) > 0. 
Furthermore, [^^{"^w^"'^^ is in ^^(0, 1) if and only if ±^{S) > 0. 

4.2. Spectral analysis. We determine the spectrum and the spectral decomposition of T. 

For functions f,g that are differentiable at a point x E (—1, 1) we define 

[f,g]{x)=p{x)W{f,g){x), 

where 

p{x) = C{1 - xr+'{l + xf+\ C = 2— /3-ii^±^L^, 

and W{f, (?) denotes the Wronskian 

W{f,g){x)^r{x)g{x)-f{x)g'{x). 

For l<a<6<lwe denote by 'D{a,b) the subspace of L"^ (^{a,b),w^°''^\x)dx) consisting of 
functions / such that 

• / is continuously differentiable on (a, b) 

• f is absolutely continuous on (a, b) 

• Tf € L^{{a,b),w<'°''^\x)dx) 

Note that V{a,b) is dense in L'^{{a,b),w^°''^\x)dx) . 
Lemma 4.8. Let 1 < o < 6 < 1 and f,gG ©(a, b), then 

((r/)(:r)^- /(x)(T^) = [f,gKb) - [f,g]{a). 



f 

J a 



Proof. We write the differential operator T as 

T = (1 - + a;)-''^ ((1 - xT+^l + + p(l - 

then it follows that 

({Tf){x)^-f{x)iTg)^) w^^^f'\x)dx = 

dx. 

~ ' / " ' ' da; V" ' ~ ' / _ 
Using integration by parts this is equal to 



x') 



g{x)^{p{x)f'{x)) - f{x)^{p{x)g'{x)) 



P{x)f{x)g{x) -p(x)f{x)g'{x) , 

J a 

which gives the result. □ 
Let Vq gT-L consist of the functions in 1, 1) with support on a compact interval in (—1, 1). 
Proposition 4.9. The densely defined operator [T^Vq) is symmetric. 

Proof. Clearly we have lima:4,_i[/, = lima;-|-i [/, ^] (x) = for f, g £ Vq. Then the result follows 
from Lemma 4.8. □ 
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The function x [/, g'](a;), x G (—1, 1), is constant if / and g are solutions of the eigenvalue 
equations Ty = \y. In the following lemma we determine the value of the constant in case of the 

eigcnfunctions ip^ on (fy^ . 

Lemma 4.10. For XeC\ (-oo, -(a + l)^), 

['^A'^a] = -^-^ bPt^(l^t] = -vDc{-7];S), 

where D = 2"+l^+^C. 

Note that [ip'^,tp'j^] and [-0^,0^] can be obtained from Lemma 4.10 using {a,(3,x) i-)- {(3,a,—x) 
and {6,ri) {—6,—r]), respectively. 

Proof. We have 



Using 



{x) = --{(3tv + 1)(^) (i + 0{1+x)), xi-1, 



dx 
we find 

Now from the connection formula (4.1) wc obtain 

0+] = c(-77; S) [cf,- , <^+] = -r^D c(-7?; S) . □ 

Let us mention that from the explicit formula (2.4) for c{—r]; 6) and Lemma 4.10, it follows that 
[V'^, = if and only if A e C \ (— oo, —{a + 1)^) is a solution of 

(4.3) i(l + (5(A)+7?(A)±K) =-n, n e N, 

or equivalently, for some n G N, 



(4.4) y/X + ia + 1)2 + y/X + {j3 + 1)2 = -2n~lTK- 

Proposition 4.11. The symmetric operator [T.Vq) has a unique self-adjoint extension. 

Wc denote the self-adjoint extension again by T. 

Proof By [4, Thm. Xin.2.10] the adjoint of {T,Vq) is (T, 2?(-l, 1)), so the deficiency spaces 
of (T, I?o) consist of the solutions of the differential equations Tf = ±if that are in [4, Cor. 
XIII.2.11]. Let / G "H be a solution of Tf = if , then / must be a linear combination of (f>f and 
since these are linearly independent solutions of this eigenvalue equation by Lemma 4.10. Note 
that SR(?7(i)) > 0, so by Remark 4.7 (f)~ is not near —1, which implies that / is a multiple of (pf. 
In the same way it follows that / is a multiple of tjj^ . But (f)'^ and ^jj^ are linearly independent 
by Lemma 4.10, hence / = 0. In the same way it follows that f & H satisfying Tf = —if is 
the zero function. So T has deficiency indices (0,0), which implies it has a unique self-adjoint 
extension. □ 

Assume A e C \ R. In this case ^{r]),^{S) > 0, so (j)^ e L'^{{-l,0),w^"^l^\x)dx) and 
e I'^((0, l),w^"'^\x)dx). We define the Green kernel by 

r0+(x)^+(z/) 



Kx{x,y) 



x<y, 



X > y, 



then K{-,y) e H for any y £ (—1,1). The Green kernel is useful for describing the resolvent 
operator Rx = iT-X)-'^. 
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Lemma 4.12. For A e C \ M the resolvent R\ is given by 
Proof. First note that if / G Pq, 



lim [</>+, /](x)=0 = lim[V'+,/](x). 
x4.— 1 xtl 



Now from Lemma 4.8 we obtain 



Rx{T-X)f{y) = -^^iM^ r {{T-X)f){x)ct>+{x)w("'^\x)dx 



-'X ' V^A 

since (T — A)^^ = and (T — A)^^ = 0. Writing out the terms between brackets, we obtain 
Rx{T - X)f{y) = ( - fiy)piy)^p+iy)^{y) + f(y)p(y)cl,+ (y)^{y)) = f{y). 

Now we can determine the spectral measure of T by 

(4.5) {E{a,b)f,g) = \un\im^ f ' ({Rx+iJ, g) - {Rx-iJ, g))dX, f,gGVo, 
see [4, Thm. Xn.2.10]. We write 

(4.6) {Rxf,9)= II {f{xM + fiy)W)) "^^"'^^ (^)^^"'^^ (y) d{x, y), 



^/ ((r-A)/)(x)V+(x)«;("'«(x)cix 
^-^^ {y) [/. 0+] ( jy) - 0+ (y) [/, ] {v)) , 



where A = {{x, y) & \ -I < x <l, x <y <l}. 

Let A e M. To compute the spectral measure we have to consider the limit 

(47^ <I^Ue{^)i^Ue{y) <I^UM^Ue^y) 



eio \ r]{X — ie)c{—ri{X — ie); S{X — ie)) r]{X + ie)c{—r]{X + ie); 5{X + is)) j 
Note that 



'limr/(A-ie) eM>o, if A + (/3 + 1)^ > 0, 
£4-0 



limr/(A + ie) = < 
£4-0 



limr/(A-i£) eiM>o, if A + (/3 + 1)^ < 0, 
£4-0 



and 



'lim5(A-i£) eM>o, if A + (a + 1)^ > 0, 
£4-0 



lim5(A + ie) 
£4-0 



lim(5(A- ie) G «>o, if A + (a + 1)^ < 0. 

. £^0 



We see that we have to distinguish several cases. 

4.3. The continuous spectrum. We define an integral transform J^i^^ mapping f G Vq to 
C^-valued function on 1^2 = (— oo, — (/3 + 1)^) by 



(^fV)(A) = Ij{x) (^^g)^(-«(,) 
where the functions ip"^ are defined by (2.10). 



dx, X e O2, / e Vq, 
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Proposition 4.13. Let a, 6 e O2 with a <b, then 



where (recall from {2.7)) 

V21W - 



c(-??a;5a)' 



and vi2{X) = U2i(A). Here 5x and r]\ are defined by (2.2). 

Proof. First observe that lim^j.o'^lA + ie) = 5\ and liiriej.o'^lA — ie) = S\ = —6x. Similarly, 
lim^^o r/(A + is) = rjx and limeio r/(A -is) =r]^ = -rjx. This gives us 

hm <^J+fe = <pj , hm <^J_fe = <Pa • 
£4-0 £4-0 



Now the limit in (4.7) is equal to 



Ix{x,y) =\im\ ; + f 

£■1-0 y-vxc{-vx;S\) -Vxc{vx;-Ox) J 

Using the connection formula (4.1) this becomes 
Ix{x,y) = 



which is manifestly symmetric in x and y. Using iPx{x) = fx (^)> see that 



Vx\y^xiy)J V^2i(A) 1 J \ip^{x)^ 

Now we can symmetrize the double integral in (4.6) again, and then the result follows from 

(4.5). □ 

Next we define an integral transform Tc^^ mapping Vq to complex- valued functions on ili = 
(-(/3 + 1)2, -(a + 1)2) by 

(J-(i)/)(A) = j^J{x)<fx{x)w^''^f'\x)dx, A G 1^1, / e Do, 

where ipx{x) is defined by (2.9). 

Proposition 4.14. Let a, 6 e Oi with a <b, then 



{E{a,b)f,9) = [\j'Pf){X){J'Pg){X)v{X) 



dX 

where (recall from (2.5)^ 

""^^^ = c{5x;n{>^))c{- 5x;nWy 

Proof. In this case 

lim5(A + ie) = 5a = — liin'5(A — ie) and lim?7(A + ie) = 77(A) = limr;(A — ie). 
£4-0 £4-0 e4.o £4.0 

Consequently, using Euler's transformation. 



and 



lim (tit , . = ipx = lim (bf ■ , 

£4-0 £4-0 



limthf .(x) = lim V'f I ,v(a;)- 
£^0 ^^-»£^ ^ £4.0 ^ 
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SO that the limit (4.7) is equal to 



hix,y)=liml ^7 V/^, : + 



where we have used Lemma 4.4(i). Using the connection formula (4.2) we obtain 



Ix{x,y) = lim 



eio -5xc{6x;r]W)c{-Sx;vW) 

^ 'P\{x)fx{y) 

-Sxc{6x;vW)c{-dx;vW)' 
The result follows from (4.5) and (4.6) after symmetrizing the double integral. □ 

Since the spectrum is a closed set, the points —{a + 1)^ and —{(3 + 1)^ must belong to the 
spectrum. 

Proposition 4.15. The points —{a + 1)^ and — (/3 + 1)^ belong to the continuous spectrum. 

Proof. This follows from the fact none of the eigenfunctions is in H for these values of A, see 
Remark 4.7. □ 

4.4. The discrete spectrum. Recall the finite set fi^ = {A„ | n e N and n < |(k — 1)}, where 
A„ is defined by (2.3). For k > 1, i.e., if fid is nonempty, we define the integral transform J^a on 
^by 

{T,.f)iX) = {.f.^x), AeC!d, /G^, 
where ipx^ is defined by (2.11). Note that tpx^ = 4'x ■ 

Proposition 4.16. Let —{a + 1)^ < a < b. IffldC^ (a, b) consists of exactly one number A„, then 
{E{c 

where (recall from (2.8)) 



{E{a,b)f,g} = (J-rf/)(A„)(J-d3)(A„)^, f,g€n, 



A=A. \r,iX)c{-rj{Xy,5{X)) J ' 
Furthermore, if fid fl (a, b) = 0, then 

{E{a,b)f,g) = 0, f,g€n. 
Proof. Assume Cla H {a,b) = {A„}. By (4.5) and (4.6) we have 

{E{a, b)f,g)=D-^ jj [f{x)W) + fiy)9{^) w^^^'^^ (x)u;("'« {y) 

1 r ,,+ ! y\, 1 

d{x,y), 



J_ [ ^^A^fMlll^_dX 
2niJc -v{X)c(-v{X);6{X)) 



{-v{xy,6{x)) 

where C is a small clockwise oriented rectifiable closed curve encircling A„ exactly once. The 
integral over the curve C is equal to 

't>xAx)^xAy) ^ 



^=A„ \n{x)c{-r,ixy,six)) J ' 

By (4.3) we have c(— 77(A); 5(A)) = if and only if A = A„, n G Z, where A„ is defined by (2.3). 
So in this case (4.1) becomes tpl^ = c{r]{Xn); (5(A„))(/)^ , from which we see that the integrand is 
symmetric in x and y. Symmetrizing the double integral gives the result. □ 

Corollary 4.17. Suppose fla is nonempty, then the following orthogonality relations hold: 

{^Xm^VXn) = — ^mn, A^, A„ G firf. 
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Proof. Let A„,Am € Qd and set / = ipx^ and g = ip\^. From Proposition 4.16 it follows that 
{fXm ; "y^An) = if A„ ^ Ato. Furthermore, if A„ = A^, then 

from which the result follows. □ 

We have now completely determined the spectrum of T. 

Theorem 4.18. The self-adjoint closure of the densely defined operator (T, I?o) has continuous 
spectrum ( — oo, — (a + 1)^] and (possibly empty) discrete spectrum Q4. The sets 0,2 andQi inside 
the continuous spectrum have multiplicity two and one, respectively. 

4.5. The integral transform. We define an integral transform T on T>o by 

(4.8) J-/ = T^f + :Fi^^f + :Faf, f e Vo. 

For f G T>Q this coincides with (2.12). 

Proposition 4.19. J" extends uniquely to an isometry T :'H ^ -^^(^)- 
Proof. For f,g&T>o we have 

(/,5) = (^/,^5)v 

by Propositions 4.13, 4.14 and 4.16, so : Vq ^ L^O^) is an isometry. By continuity it extends 
uniquely to an isometry H — )• i^(V). □ 

Our next goal is to show that F : % ^ -^^(^) is surjective and determine the inverse. For 
convenience we assume that T has no discrete spectrum. 
For < a < 1 we define 



{f,9)a= r f{x)g{x)w^"'^\x)dx, 

J —a 



for all functions /, g for which the integral converges. Note that for f,gGH the limit a f 1 gives 
the inner product (/, g). Suppose now that f\ is a solution of the eigenvalue equation Tf\ = Xfx, 
then by Lemma 4.8, 

/f f ^ [fxJx'Ka) - [fx,fx']{-a) x y ^ ro 
\Jx,JX')a = X- X' ' ' 

We will use this expression with fx = ip^ and we want to let at 1- We need to consider several 
cases. 

4.5.1. Case 1: A, A' e ^2. From Lemma 4.6 we find for x i —1, cf. the proof of Lemma 4.10, 

D, /i + ^\i(''^-''v) 



H^VxM^) = y (^A + VX-) [-^J (1 + 0{l + x)) , 

, vl,]{x) = -ivx - VX') i^^j (1 + 0{1 + x)) . 



The behavior at a; = —1 of [(p^ , (p~^,]{x) and [ip^ , 'Py]{x) follows from (p'l{x) = (f^ (x). For x t 1 
we use the expansion from (4.2) (recall that = limg^o 4'x+ie) Lemma 4.6 to find 

K,<^A'](^) = f E <^^y,Vx)c{-^6xr,-r,x'){e6x+e'6x')(^y^ ' ''\l+0{l-x)), 



and 

ivh^U^) = Y E c{e6x;vx)ci-e'6xr,vx'){edx + e'6x>)(^) (l + (!?(l-x)). 

We will need the following behavior of the c- functions. 
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Lemma 4.20. The c-function defined by (2.4) satisfies 

'©(e-'^^), A-^-oo, 



c{-6x;v\) 



[0{1), At-(/3 + l)^ 

fo(l), A-s^-oo, 
\0{1), At-(/3 + l)^ 



Proof. This follows from the definition (2.4) of the c-function, and well-known asymptotic prop- 
erties of the F-function, see e.g. [2, Section 1.4]. □ 

Proposition 4.21. Let f e (7(^2) satisfy 

' \o{i), At-(/3 + l)^ 

for some s > 0, and let X' then 

lim / f{X){tpl,<py)adX = -2'KiDSx>c{-Sxr,r]x>)c{6xr,-r]x>)f{X'). 
Proof Note that 

,^4 9) + VX' _ 1 V\- VX' _ 1 



A - A' vx-Vx'' A - A' r]x+ rjy ' 



and similar expressions are valid for 5x- Now use 5x = i\5x\ and r]x = and write A'' = 

-iln(if^), then 

2 JV->oo Jq^ ee{+-} ^ ^ ^' ^ ^ ^' 

lim / ;(,)CO^^(I^^IHryv|),,_ig ^ (,)SiniV(|,.|-|.v|) 

where 

^±(A) = c{6x; Vx)c{e6x';-vx') ± c(-(5a; Vx)c{-edx';-vx')- 

The terms with vanish by the Riemann-Lebesgue lemma, which follows from Lemma 4.20 and 

the assumptions on /. 

Claim: 

lim / /(A)r(A) ^°^^i''^^'"'^^''^ dA- lim / fixf-^^^^iM^^dX = 0. 

Proof of claim. Using (4.9) and cos 61 — cos ^2 = — 2sin(^^-j^) sin(^^-^), we obtain 

,~^v, cosN{\Sx\ - \6x'\) cosjV(|r?;,|-|7?v|) CiX){6x + Sx') - {yx + Vx') , ,^ 

?-(A) J J = r r; cosA/dcDA - ^a' ) 

dx - Ox' rjx -Tjx' A - A' 



+ 



2 sin f (1^,1 + |r?,| - l^vl - |7?v|) sin f (|(5a| - |^a| + \5x'\ - M 

Vx - rjx- 



We multiply the right hand side of the above identity by /(A) and integrate over A. Since the 
function A i-^- inx+nx') ^ removable singularity by Lemma 4.4(ii), it follows from 

the Riemann-Lebesgue lemma that the first term vanishes as — )• 00. For the second term we 
may use 

s iu^{\5x\ + \vx\-\5x'\-\m'. <^ 
'7A - rix' ^ 
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for A in a neighborhood of A' and for some B > 0, then we see that we can apply the Riemann- 
Lebesgue lemma again, which proofs the claim. □ 

To finish the proof of the proposition we use 

sin N{x — y) 



(4.10) 



lim — / q(x)- 



x-y 



dx = g{y), 



if g G {A, B) is continuous. Then 



iD 



lim / f{\){^l,vl.)^d\= — y,va^ / /(A)^+(A) 



sinAr(|<5A|-|<5v|) 



5x - 6y 



dX 



? lim / /(A) 

2 N^ooJa/^ ' 



sin7V(|r/A|- |r?A'|) 



dX 



Vx - VX' 



= -niD{6y^^{X')+vx')f{X') 



provided / and / arc c;ontiimous functions in L^{fl2), which is indeed the case. Here we used 

the substitutions a; = and x — \rix\ before applying (4.10); note that ^ = in both cases. 
Finally, applying Lemma 4.4(ii) with {x,y) = {Sx,rix) the last expression becomes 



-27nDdx'c{-6x';r,x')c{6x';-r]x')f{y), 

which finishes the proof. 

The following result is proved in the same way as Proposition 4.21. 



□ 



Proposition 4.22. Let f e C{Q2) satisfy the same conditions as in Proposition 4-^1 o-nd let 
X' & ^l2, then 

lim [ f{X){ip+,ip+}adX = -2niDdx'c{6x';r,x')c{-6x';r,x')f{X'). 
«Ti Jn2 

By combining Propositions 4.21 and 4.22 we obtain the following result. 
Proposition 4.23. Let f\ and satisfy the conditions from Proposition 4-21, then 



2ttD J a, \V>x ix)J \hW 



dX 



(A') = -rSx,AiX') (/^(^^;) 



where 



A(X')=( '^(~'^^';'?^')^(^^'!~'?^') c{h';vx')c{-6x';vx') 
\c{Sx';-Vx')c{-Sx';-r)x') c{-6x';vx')c{Sx';-r)x') 



Proof. Let /i and /2 satisfy the conditions of Proposition 4.21, then from this proposition and 
from applying Fubini's theorem we obtain 

-iSyc{Sx';vx'H-Sx';-r]x')f2{X') = :r^lim / /2(A) P ip+{x)ip-^,{x)w('''^\x)dxdX 

ZTTU atl J -a 

ipy{x)w'^"''^\x)dx. 



From Propositions 4.21 and 4.22 we find three similar identities, leading to 

'^^ I / / fnj I T \ \ / r-. ( A 1 \ 

dX 



1-1 >D In 



^l{x)\ //i(A) 
nA^'x^^)) V/2(A) 



gg) .(-)(.).. = -...M(A')(;:[J)), 



which is the desired result. 

We need the inverse of the matrix A{X) from Proposition 4.23. 
Lemma 4.24. For X e Q.2, A{X)-'^ = V{X) with V{X) defined by (2.6) 



□ 
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Proof. We have 

det ^(A) = c{5y, -rjx)c{-Sx; r]x) (c{Sx; -r]x)c{-6x; V\) - c{Sx; ?7a)c(-5a; -??a)) 

= Y^c{Sx;-V\)ci~Sx]r]x), 
Ox 

by Lemma 4.4(ii). Now it is straightforward to compute the inverse of A. The result then follows 
from the definition of V{X) and Lemma 4.4(i). □ 

Let Co{^2', C^) denote the set of continuous C^-valued functions g = {$1) on f22 satisfying 

rO(|A|-^-), A ^-00, ._ 

^^^^^ \o{\x+{p+im, At-(/3+l)^ ^ ' ' 

for some £ > 0. For g e Co{rt2', C"^) we define the function Si^^g by 

Proposition 4.25. Let g G Co{Sl2;C'^) and A e fl2, then {J^Poi^^ g){X) = g{X). 

Proof. Let g € Co{fl2;C'^), define the C^-valued function / by /(A) = ( JJ[a)) = z^5^M>')~'^9W- 
Since 

'©(e^'^^), X^-oo, 

0{1), At-(/3 + l)2, 

by Lemma 4.20, the functions /i and /2 satisfy the conditions from Proposition 4.21. Now Propo- 
sition 4.23 shows that 



V2l{X) 









2 V^A ix)J 



l^^A{X)-'g{X)dX 



(A')=5(A'). 



(2) 

Prom Lemma 4.24 we see that the term inside square brackets is exactly {Gc 9)ix). □ 
4.5.2. Case 2: A, A' e Qi. In this case, 



lim [(px-,V\']{x) = 0, 



and for X 1 1 we have 

[<fx,fx']{x) = 



^ J2 c{eSx;vW)c{-e'Sx'-Mm^Sx + e'6x')(^^y^''' ' + 0{1 - x)) . 



e,e'e{ + ,-} 

We have the following behavior of the c-functions. 
Lemma 4.26. The c-function defined by (2.4) satisfies 

In the same way as in Proposition 4.21 this leads to the following result. 

Proposition 4.27. Let f be a continuous function satisfying 

^(0{1), Xi-{f3 + l) 

\0(|A + (a + l)2|-H^), At-(a + l)' 

for some £ > 0, and let X' G fli, then 

lim/^y(A)(^„^.,)„dA = -|j^/(AO, 

where (recall from (2.5); u(A') = (c((5A';^/(A'))c(-(5A';r/(A')))~^ 



/(A) = 



^2 
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Note that 

A ;-(/? + 1)2, 

\2 



v{X) 



\0{\X + {a + lf\), Xt-{a + iy 
by Lemma 4.26. Let Co(S^i) denote the set of continuous functions g on fli satisfying 



5(A) 



o(i), a;-(/3 + i)2, 

0{\X+{a + im, At-(a + l)2, 



for some £ > 0. We define an integral transform Qc^^ on Co(fii) by 

{Gi'^9){x) = ^ ^ g{X)vx{x)W^'H^)^^, X e (-1, 1), g G Co(Oi). 
Now similar as in Proposition 4.25, it follows from Proposition 4.27 that J-'i^'' is a left-inverse of 



Proposition 4.28. For g e Co(Oi) anrf A e ili, we /ia?;e (J^^^^Oi^^ g){X) = g{X). 

4.6. The integral transform g. We define G on Co(Oi) U Co(1^2;C2) by ^ = G^^'' ® We 
will show that is a left-inverse of Q. We need the following result. 

Proposition 4.29. 

(i) Let A G andge Co(l^i), then (J"f ^d^^g)(A) = (g). 
("m; iei A e O2 and e Co(1^2;C2), i/ien (ji^^e?? ^5)(A) = 0. 

Proof. Let A e fi2 and A' e Oi, then 

li™ bA'</^A'](a;) = 0, 
x4-— 1 

and for x t 1, 

[<^J,<^A'](a;) = 

^ ^ c(e(5A;±r,A)c(-eV;r?(A'))(e<5A + e'(5A')(^)'^''' ' '"^ (l + 0(1 - x)) . 

Similar as in the proof of Proposition 4.21 it follows by application of the Riemann-Lebesgue 
lemma that 

lim / f{X){ipf,ipx')adX = lim .f{X)^ r -f dX = 0, 

ati ia^ ati Jq2 a - A' 

for suitable functions /. As in Proposition 4.23 we obtain from this {Tc^^Gc^^ g){X) = (o). 
In the same way, it follows from 

lim / /(A)(^„^J,).ciA = lim/ /(A) t^^' ' t^^^' ^"'^^-^^ dA = 0, 

afl an Jq^ a - A' 

for suitable functions /, that {J^c^^Gc^^ g){X) =0. □ 
Combining Propositions 4.25, 4.28 and 4.29 shows that {FoG)g = 5 for 5 e Co(Oi)uCo(02; C^). 

Proposition 4.30. The integral transform G extends uniquely to an operator G ■ -t'^(V) — )• H such 
that G = J"^^- 

Proof Let g € Co(fii) U Co(1^2; C^), then 

{g,g)v = {{^ °G)g,{J^ oG)g)v {Gg,Gg), 

by Proposition 4.19. Since Co{^i)UCo{fl2; C^) is dense in H, G extends by continuity uniquely to 
an operator G '■ ^^(V) H, and ToG extends to the identity operator on H, hence G = . □ 
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Remark 4.31. In case the discrete spectrum is nonempty the inverse of is the extension of 
the operator Q = ^i^^ ® Oi^^ Qd with 

{gd9){x) = ^Y1 9W^x{x)Nx, X e (-1, 1), 

for any function (7 : — ^ C. The proof in this case is the same as in the case of empty discrete 
spectrum. 
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